
from pylab import *
#import numpy as np
#import matplotlib.pyplot as plt

from numpy import linalg as la


train_5 = np.loadtxt('train_5.txt',delimiter=',')   # 556 samples
train_6 = np.loadtxt('train_6.txt',delimiter=',')   # 664 samples


# Digit 5
n=9;
I = np.reshape(train_5[n,:],(16,16))

plt.imshow(I,cmap='gray')
plt.show()


# Digit 6
n=5;
I = reshape(train_6[n,:],(16,16))

plt.imshow(I,cmap='gray')
plt.show()


x_train_brut = np.vstack((train_5[:145,:], train_6[:200,:]))
N_train = np.size(x_train_brut,axis=0)
class_train = np.ones((345,1))   # label 1 for digit 6
class_train[:145] = 0       # label 0 for digit 5

x_test_brut = np.vstack((train_5[145:,:], train_6[200:,:]))
N_test = np.size(train_5,axis=0)+np.size(train_6,axis=0)-N_train


# Principal component analysis
moy_train = x_train_brut.mean(axis=0)  # all the data, 5 & 6
x_train_centre = x_train_brut-np.tile(moy_train,(N_train,1))

# PCA from scikit-learn
from sklearn.decomposition import PCA
pca = PCA()
pca.fit(x_train_centre)  # you may forget centering that is done by sklearn PCA

singval = pca.singular_values_   # eigenvalues
comp = pca.components_           # principal components
proj = pca.transform(x_train_centre)  # computes the projection coefficients


I_moy = np.reshape(moy_train,(16,16))   # averaged image = mean 
plt.imshow(I_moy,cmap = 'gray')
plt.show()


n=12   # choice of image no n=12 or any other

I = I_moy + proj[n,0]*np.reshape(comp[0,:],(16,16))  # adding the 1st PCA component 
plt.imshow(I,cmap='gray')
plt.show()


"""
Following the SK-learn documentation on PCA, we can say that the "Parameters" are considered as
input parameters and the "Attributes" are considered as output parameters.
Therefore, we can say that, the following are considered as input parameters:
- Number of components "n_components"
- Singular Value Decomposition solver "svd_solver"
- Tolerance for singular values computed by svd_solver "tol"
- Number of iterations for the power method computed by svd_solver "iterated_power"
- Additional number of random vectors to sample the range of X to ensure proper conditioning "n_oversamples"
- Power iteration normalizer for randomized SVD solver "power_iteration_normalizer"
- "random_state" for reproducibility of the results when the ‘arpack’ or ‘randomized’ solvers are used


And, the following are considered as output parameters:
- Principal axes in feature space "components_"
- Amount of variance explained by each of the selected components "explained_variance_"
- Percentage of variance explained by each of the selected components "explained_variance_ratio_"
- The singular values corresponding to each of the selected components "singular_values_"
- Per-feature empirical mean, estimated from the training set "mean_"
- The estimated number of components "n_components_"
- The estimated noise covariance following the Probabilistic PCA model "noise_variance_"

"""

'\nFollowing the SK-learn documentation on PCA, we can say that the "Parameters" are considered as\ninput parameters and the "Attributes" are considered as output parameters.\nTherefore, we can say that, the following are considered as input parameters:\n- Number of components "n_components"\n- Singular Value Decomposition solver "svd_solver"\n- Tolerance for singular values computed by svd_solver "tol"\n- Number of iterations for the power method computed by svd_solver "iterated_power"\n- Additional number of random vectors to sample the range of X to ensure proper conditioning "n_oversamples"\n- Power iteration normalizer for randomized SVD solver "power_iteration_normalizer"\n- "random_state" for reproducibility of the results when the ‘arpack’ or ‘randomized’ solvers are used\n\n\nAnd, the following are considered as output parameters:\n- Principal axes in feature space "components_"\n- Amount of variance explained by each of the selected components "explained_variance_"\n- Percentage of variance explained by each of the selected components "explained_variance_ratio_"\n- The singular values corresponding to each of the selected components "singular_values_"\n- Per-feature empirical mean, estimated from the training set "mean_"\n- The estimated number of components "n_components_"\n- The estimated noise covariance following the Probabilistic PCA model "noise_variance_"\n\n'


n=12   # choice of image no 12
kmax= 255  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax+1):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
    plt.imshow(I_moy + update,cmap='gray')
    plt.show()
    print("The reconstruction of digit 5 adding ", k+1," principal components")

The reconstruction of digit 5 adding  1  principal components

The reconstruction of digit 5 adding  2  principal components

The reconstruction of digit 5 adding  3  principal components

The reconstruction of digit 5 adding  4  principal components

The reconstruction of digit 5 adding  5  principal components

The reconstruction of digit 5 adding  6  principal components

The reconstruction of digit 5 adding  7  principal components

The reconstruction of digit 5 adding  8  principal components

The reconstruction of digit 5 adding  9  principal components

The reconstruction of digit 5 adding  10  principal components


n=12   # choice of image no 12
kmax= 15  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 5 adding ", k+1," principal components")
    
"""
Adding 15 principal components is giving an acceptable reconstruction of the digit 5.
"""

The reconstruction of digit 5 adding  15  principal components

'\nAdding 15 principal components is giving an acceptable reconstruction of the digit 5.\n'


n=12   # choice of image no 12
kmax= 29  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 5 adding ", k+1," principal components")
    
"""
Adding 29 principal components is already giving  a nice reconstruction of the digit 5.
"""

The reconstruction of digit 5 adding  29  principal components

'\nAdding 29 principal components is already giving  a nice reconstruction of the digit 5.\n'


n=12   # choice of image no 12
kmax= 53  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 5 adding ", k+1," principal components")
    
"""
Adding 53 principal components is giving a very nice reconstruction of the digit 5.
"""

The reconstruction of digit 5 adding  53  principal components

'\nAdding 53 principal components is giving a very nice reconstruction of the digit 5.\n'


n=200   # choice of image no 200 (because it is between 145 and 345)
kmax= 255  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax+1):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
    plt.imshow(I_moy + update,cmap='gray')
    plt.show()
    print("The reconstruction of digit 6 adding ", k+1," principal components")

The reconstruction of digit 6 adding  1  principal components

The reconstruction of digit 6 adding  2  principal components

The reconstruction of digit 6 adding  3  principal components

The reconstruction of digit 6 adding  4  principal components

The reconstruction of digit 6 adding  5  principal components

The reconstruction of digit 6 adding  6  principal components

The reconstruction of digit 6 adding  7  principal components

The reconstruction of digit 6 adding  8  principal components

The reconstruction of digit 6 adding  9  principal components

The reconstruction of digit 6 adding  10  principal components


n=200   # choice of image no 200
kmax= 1  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 6 adding ", k+1," principal components")
    
"""
Adding 1 principal components is already giving  an acceptable of the digit 6.
"""

The reconstruction of digit 6 adding  1  principal components

'\nAdding 1 principal components is already giving  an acceptable of the digit 6.\n'


n=200   # choice of image no 200
kmax= 25  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 6 adding ", k+1," principal components")
    
"""
Adding 25 principal components is already giving  a nice reconstruction of the digit 6.
"""

The reconstruction of digit 6 adding  25  principal components

'\nAdding 25 principal components is already giving  a nice reconstruction of the digit 6.\n'


n=200   # choice of image no n=200 or any other
kmax= 80  # Number of principal components to add successively
update = np.zeros([16,16])

for k in range(0,kmax):
    update = update + proj[n,k]*np.reshape(comp[k,:],(16,16)) # adding the PCA components
plt.imshow(I_moy + update,cmap='gray')
plt.show()
print("The reconstruction of digit 6 adding ", k+1," principal components")
    
"""
Adding 80 principal components is already giving  a nice reconstruction of the digit 6.
"""

The reconstruction of digit 6 adding  80  principal components

'\nAdding 80 principal components is already giving  a nice reconstruction of the digit 6.\n'


import matplotlib.pyplot as plt

# Separate the x and y coordinates
x_coords_5 = proj[0:145,0]
y_coords_5 = proj[0:145,1]

x_coords_6 = proj[145:345,0]
y_coords_6 = proj[145:345,1]


# Plot points with two different colors
plt.scatter(x_coords_5, y_coords_5, color='blue', label='Digit 5')
plt.scatter(x_coords_6, y_coords_6, color='red', label='Digit 6')
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.title('Cloud of Points associated to digits 5 and 6')
plt.legend()
plt.show()


"""
By selecting to plot only the 2 first components of PCA, we could visualize the diffirence of vaiances between the
5 and 6 digits.
The two clouds of points of digits 5 and 6 are therefore linearly separable from this scatter plot.
"""

'\nBy selecting to plot only the 2 first components of PCA, we could visualize the diffirence of vaiances between the\n5 and 6 digits.\nThe two clouds of points of digits 5 and 6 are therefore linearly separable from this scatter plot.\n'


"""
YES. PCA step makes possible the use of much simpler classification since it gathers the most variations of the data in the
first components of the projection.
"""

'\nYES. PCA step makes possible the use of much simpler classification since it gathers the most variations of the data in the\nfirst components of the projection.\n'


"""
We can use PCA in combination with LDA.
Using only the 2 components of PCA, we can linearly separate the digits 5 from 6.

Note that LDA is more sentive to outliers than logistic regression. In practice, we will advise to use logistic regression
for this task.
"""

'\nWe can use PCA in combination with LDA.\nUsing only the 2 components of PCA, we can linearly separate the digits 5 from 6.\n\nNote that LDA is more sentive to outliers than logistic regression. In practice, we will advise to use logistic regression\nfor this task.\n'

TNE 2: Principal Component Analysis¶

1. Application: handwritten digits recognition 5 & 6¶

Examples of images:¶

Separating the training and test sets¶

2. Principal Component Analysis¶

Documentation¶

Display the averaged images of 5 & 6 respectively¶

Display an example rebuilt from the 1st component only¶

Exercise 1: PCA & approximation¶

1. Identification of input and output parameters as per the documentation¶

2. Progressive reconstruction of an image of digit 5¶

Adding the successive contribution of principal components¶

3.1. How many components are necessary to obtain a reconstruction considered as acceptable?¶

3.2. How many components are necessary to obtain a reconstruction considered as nice?¶

3.3. How many components are necessary to obtain a reconstruction considered as very nice?¶

4. The same analysis for 6¶

4.1. Adding the successive contribution of principal components¶

4.2. How many components are necessary to obtain a reconstruction considered as acceptable?¶

4.3. How many components are necessary to obtain a reconstruction considered as nice?¶

4.4. How many components are necessary to obtain a reconstruction considered as very nice?¶

Exercise 2: PCA & classification¶

1. Display the cloud of points associated to digits 5 and 6 by using 2 different colors.¶

2. Comment on the repartition of points in the plane.¶

3.1. Do you see how this PCA step makes possible the use of a much simpler classification?¶

3.1. What would you propose as an alternative to logistic regression of TP3 then?¶

Beyong this lab¶